Evaluation Metrics Trade-off for Recommendation Systems

Following the famous Netflix Prize competition, accuracy metrics (such as RMSE) became a de facto standard in the community to evaluate and compare performance of recommendation algorithms. However, it has been shown, that accuracy is not the best way to quantify the quality of recommendations, especially in the context of predicting online performance from offline evaluations [1]. Thus, it makes sense to look at other metrics, such as novelty, coverage, or serendipity, and in particular, trade-off between these metrics [2]. In this project, a student will work with large news recommendation datasets, collected online on swissinfo.ch and lepoint.fr. The goal would be to implement a number of recommender algorithms with tunable hyperparameters, and generate trade-offs between different evaluation metrics by varying these hyperparametrs. A student will then analyse these trade-offs by plotting trade-off curves, comparing them for various algorithms, examining the area under the curves, and possibly looking at their impact on CTR and online success rate.